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Abstract 

Training and coaching with performance feedback has been effective for supporting teachers to use evidence-based 
instructional practices. However, coaching with performance feedback has primarily been used to support teachers to 
use discrete skills, and there has been little evidence of maintenance and generalization. The purpose of this study was 
to examine the effects of a professional development intervention on teachers’ implementation of practices related to 
the Pyramid Model for Promoting Social-Emotional Competence in Young Children, as well as the extent to which teachers 
generalized and maintained those practices. A multiple probe design across sets of Pyramid Model practices replicated 
across three teachers was used in this study. All teachers acquired the practices and maintained the practices after 
coaching ended. There was some evidence of generalization for all three teachers. The effects of teacher implementation 
on classroom-wide incidences of challenging behavior were mixed. Teachers all rated the coaching positively. 
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Recent reviews of the literature indicate that the prevalence 
rates of challenging behavior in preschool settings are 
between 14% and 34% (Qi & Kaiser, 2003; Upshur, Wenz- 
Gross, & Reed, 2009). Without intervention, children with 
challenging behavior may have difficulty interacting with 
their peers and may fall behind in academic skills (Gilliam, 
2005). Challenging behavior can lead to children being 
expelled from preschool programs, further isolating them 
from effective interventions and peers with whom to inter¬ 
act. Gilliam (2005) conducted a survey of educators in 40 
states and found the preschool expulsion rate to be approxi¬ 
mately 6.67 per 1,000 preschool children enrolled. This rate 
is 3.20 times higher than the expulsion rate of students in 
K-12th grades (Gilliam, 2005). 

Given these data, it is clear that early childhood educa¬ 
tors need support around addressing the needs of children 
with ongoing challenging behavior. Early childhood educa¬ 
tors have consistently identified addressing the needs of 
children with challenging behavior as one of the major chal¬ 
lenges of their job and thus as a primary training need 
(Friedman-Krauss, Raver, Neuspiel, & Kinsel, 2014; Snell 
et al., 2012). Furthermore, teachers who report having 
access to behavior support personnel are less likely to ask 
for children to be removed from their classrooms (Gilliam, 


2005). The provision of training for early childhood educa¬ 
tors and access to behavior supports is consistent with 
implementation science, which suggests that it is the combi¬ 
nation of effective intervention practices and effective 
implementation practices that lead to positive outcomes for 
children and families (Fixsen, Blase, Duda, Naoom, & Van 
Dyke, 2010). 

Intervention Practices: Pyramid Model 
for Addressing Social-Emotional 
Competence 

The Pyramid Model for Promoting Social-Emotional 
Competence in Young Children (Fox, Dunlap, Hemmeter, 
Joseph, & Strain, 2003) is a comprehensive, tiered model for 
promoting young children’s social-emotional development 
and addressing challenging behavior. The Pyramid Model 
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includes evidence-based practices designed to support 
young children’s social-emotional skills, and prevent and 
address challenging behavior in the classroom. The Pyramid 
Model includes three tiers of support. The first tier includes 
practices that are implemented universally and are related to 
creating high-quality, supportive environments and nurtur¬ 
ing and responsive relationships. Some practices associated 
with this tier of the Pyramid Model are having meaningful 
conversations with children, supporting children through 
play, designing engaging learning centers and activities, 
implementing a predictable schedule, teaching behavior 
expectations, using positive descriptive feedback, and plan¬ 
ning transitions. The second level of the Pyramid Model 
involves providing targeted social-emotional supports, 
including teaching friendship skills, emotional literacy, 
anger management, and social problem-solving. For chil¬ 
dren who need more intensive support around social-emo¬ 
tional skills, teachers implement individualized instruction 
with sufficient intensity for each child. The tertiary tier of 
the Pyramid Model involves the use of targeted interven¬ 
tions for individual children whose challenging behavior or 
other social-emotional needs are not responsive to develop- 
mentally appropriate guidance procedures. Targeted inter¬ 
ventions are developed by a team, involve functional 
assessment, and include three components: prevention strat¬ 
egies, teaching new skills, and responding in ways that sup¬ 
port the child’s acquisition of those new skills. The 
effectiveness of the Pyramid Model in supporting all chil¬ 
dren in an inclusive setting depends on the concurrent imple¬ 
mentation of practices across all tiers. 

Implementation Supports: Training 
and Ongoing Coaching 

For interventions, such as the Pyramid Model , to be imple¬ 
mented with fidelity, implementation supports, including 
ongoing training and coaching, are needed (Fixsen et al., 
2010). A review of the professional development literature 
identified coaching with performance feedback as a suc¬ 
cessful intervention to change teacher behavior (Snyder 
et al., 2012). Performance feedback involves the use of 
information from observations of the teacher to provide 
specific feedback on the teacher’s use of targeted teaching 
practices. The impact of coaching with performance feed¬ 
back on teachers’ use of specific social-emotional teach¬ 
ing and behavior support practices in preschool classrooms 
has been examined in several published studies. In these 
studies, researchers have examined the effects of coaching 
with performance feedback on a variety of discrete prac¬ 
tices associated with the Pyramid Model, including praise 
and precorrections (Barton, Pribble, & Chen, 2014; 
Fullerton, Conroy, & Correa, 2009; Hemmeter, Snyder, 
Kinder, & Artman, 2011; Stormont, Smith, & Lewis, 
2007). 


In addition, a series of studies have examined the use of 
training followed by coaching with performance feedback 
on teachers’ use of practices across tiers of the Pyramid 
Model. Fox, Hemmeter, Snyder, Binder, and Clarke (2011) 
used a multiple probe across skills design replicated across 
teachers to investigate whether coaching with performance 
feedback was effective in increasing teachers’ use of 
Pyramid Model practices. The teachers were trained to use 
strategies associated with each tier of the Pyramid Model. 
For all three teachers, there was a functional relation 
between coaching and increased implementation of Pyramid 
Model practices. Furthermore, there was some evidence of 
maintenance of skills following the end of intervention. 

Hemmeter and colleagues (Hemmeter, Fox, & Snyder, 
2013) conducted a cluster randomized control trial in 40 
preschool classrooms in two Southern states across two 
school years. In this study, intervention teachers received a 
3-day intensive training, followed by approximately 14 
weeks of weekly in-class coaching, debriefing meetings, 
and email feedback. The results indicated statistically sig¬ 
nificant differences between control and intervention teach¬ 
ers in the use of Pyramid Model practices as measured by 
the Teaching Pyramid Observation Tool (TPOT; Fox, 
Hemmeter, & Snyder, 2014). There was also evidence that 
teachers’ use of the Pyramid Model practices was related to 
teacher-reported improvements in social skills and chal¬ 
lenging behavior as well as observed improvements in the 
social skills of children who had the most challenging 
behavior. 

Two additional studies were conducted on the effects of 
training and coaching with performance feedback on teach¬ 
ers’ use of Pyramid Model practices. In these studies, feed¬ 
back was delivered through email. Artman-Meeker and 
Hemmeter (2013) examined the effects of coaching with 
performance feedback, delivered via email, on two teaching 
teams’ use of transition preparations, rule reminders, and 
social-emotional teaching strategies in preschool class¬ 
rooms using a multiple baseline across strategies design. 
The data showed a functional relation between coaching 
and both teams’ use of transition preparations, rule remind¬ 
ers, and social-emotional teaching practices. The effects of 
the intervention on children’s challenging behavior were 
mixed, as there was a functional relation between increases 
in the teachers’ use of targeted practices and the target 
child’s behavior for only one of the two teaching teams/ 
classrooms. 

Artman-Meeker, Hemmeter, and Snyder (2014) used a 
cluster randomized control design in 33 Head Start class¬ 
rooms to examine the effects of training and distance coach¬ 
ing on teachers’ use of Pyramid Model practices. In this 
study, teachers in both groups received a 1-day training on 
Pyramid Model practices, and intervention teachers 
received coaching with performance feedback. The obser¬ 
vations occurred via video, and feedback was delivered via 
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email. The results showed some statistically significant dif¬ 
ferences between control and intervention teachers in class¬ 
room quality. Furthermore, non-experimental analyses 
suggested that intervention teachers who accessed the inter¬ 
vention (i.e., read their email feedback and viewed training 
videos) more often used more Pyramid Model practices 
compared with both control teachers and intervention teach¬ 
ers who accessed the intervention less frequently. There 
were also reductions in child challenging behavior in the 
classrooms of teachers who accessed the intervention more 
frequently. 

Although previous studies provide support for the use of 
training and coaching with performance feedback on the 
use of Pyramid Model practices, additional research is 
needed to replicate previous results and address issues 
related to intensity of coaching, maintenance and general¬ 
ization of the effects of coaching, and teacher report of sat¬ 
isfaction and feasibility of the intervention. The current 
study was designed to address these issues using a profes¬ 
sional development intervention that included training and 
coaching with performance feedback. Specifically, the fol¬ 
lowing research questions were addressed: 

Research Question 1: Is training and coaching effective 
for increasing teachers’ use of practices related to the 
Pyramid Model ? 

Research Question 2: Do teachers generalize the use of 
coached practices to activities other than those in which 
they were coached? 

Research Question 3: Do teachers maintain practices 
after coaching on those practices end? 

Research Question 4: Does implementing the Pyramid 
Model practices with fidelity decrease classroom-wide 
instances of challenging behavior? 

Research Question 5: Does implementation of the 
Pyramid Model overall improve when teachers receive 
training and coaching on specific Pyramid Model 
practices? 

Research Question 6: What are teachers’ perspectives 
of the coaching process, coaching relationship, and sus¬ 
tainability of the Pyramid Model practices? 

Method 

Participants 

Three teachers in an urban school district participated in the 
study. Teachers were recruited from the control group who 
participated in a randomized control trial conducted by the 
same research team the previous school year (Hemmeter 
et ah, 2011). Teachers from the control group who had 
expressed an interest in coaching were contacted, and three 
teachers who consented were included in the study. During 
the randomized trial, control teachers received a 2-day 


training on the Pyramid Model, approximately 6 months 
before the beginning of the current study, but did not receive 
any type of follow-up support. 

Bianca was a Caucasian female with 8 years of teaching 
experience. She had a master’s degree and certification in 
early childhood special education (ECSE) and K-12 modi¬ 
fied special education. Kendra, a Caucasian female, held a 
bachelor’s degree in ECSE and was working toward a mas¬ 
ter’s degree. Her certification was also in ECSE. She had 2 
years of teaching experience. Susan was a Caucasian female 
with a master’s degree and certification in ECSE. She had 6 
years of teaching experience. 

Setting 

The study took place in blended preschool classrooms in 
three elementary schools. All classrooms had between 14 
and 16 children, about half who had disabilities, and all 
classrooms had a lead teacher and an assistant teacher. 
Each teacher had 2 to 4 children with persistent, ongoing 
challenging behavior and a high percentage of children 
receiving free or reduced price lunch (87.5%-93.8%). 
Observations took place in the classrooms during the regu¬ 
lar school day. Coaching sessions took place in the class¬ 
room, during naptime or after school. 

Materials 

Several types of materials were used in the study. Checklists, 
described below, were the primary data collection tool. 
Personal digital assistants (PDAs) and the Multi-Option 
Observation System for Experimental Studies (MOOSES; 
Tapp, Wehby, & Ellis, 1995) were used to collect data on 
children’s challenging behavior. Each live coaching session 
was recorded using handheld audio recorders to enable col¬ 
lection of fidelity data. 

Measurement 

Data were collected on (a) teachers’ use of targeted Pyramid 
Model practices (checklists), (b) instances of classroom¬ 
wide challenging behavior (Class-Wide Challenging 
Behavior Observation Tool [CCBOT]), and (c) teachers’ 
global use of all Pyramid Model practices (TPOT). Primary 
data were collected in the morning and included a large- 
group activity, centers/small groups, and transitions. In 
addition, generalization data were collected after lunch for 
each teacher. 

The primary behaviors of interest were the teacher’s use 
of specific practices associated with the Pyramid Model. 
These behaviors were measured through the use of 
researcher-designed checklists that were based on an earlier 
version (Hemmeter, Fox, & Snyder, 2008b) of the TPOT 
(Fox et al., 2014). Nine checklists were developed, and each 
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checklist contained 7 to 10 indicators related to the practice, 
with precise criteria for receiving credit for each indicator. 
For example, for the checklist related to teaching children 
classroom behavior expectations, specific indicators 
included the following: (a) teacher provides instructions or 
reminders on posted behavior expectations to individual 
children, during play, and within small-group activities and 
(b) throughout the observation, teacher provides specific 
positive feedback to children on meeting posted behavior 
expectations. The checklist score was a percentage, which 
was calculated by dividing the number of indicators present 
by the number of indicators possible and multiplying by 
100. These data were collected approximately 1 to 2 times 
per week. During each observation, the coach collected data 
on the teacher’s current set of targeted practices. In addi¬ 
tion, the coach collected intermittent probe data on the other 
sets of targeted practices during at least 30% of data collec¬ 
tion observations. 

Children’s challenging behavior was measured using the 
CCBOT (Hemmeter, Fox, & Snyder, 2008a). The purpose 
of the CCBOT was to provide a measure of the frequency of 
challenging behavior in classrooms; however, information 
on the reliability and validity of this measure is not avail¬ 
able. Data were collected in two 45-min CCBOT observa¬ 
tions that were conducted 4 times during the study: in 
baseline and after intervention on each set of targeted prac¬ 
tices for each teacher. Data collectors used a partial interval 
recording system with 10-s intervals to record the presence 
or absence of both low-intensity and high-intensity chal¬ 
lenging behaviors by scanning the classroom and noting 
whether any challenging behavior occurred in each interval. 
Low-intensity challenging behaviors were defined as 
behaviors that distract children or a teacher from typical 
activities, routines, or instructions. Examples of low-inten¬ 
sity challenging behavior were not following directions, 
taking a toy from another child, and wandering around the 
room not engaged with an activity or routine. High-intensity 
challenging behaviors were defined as behaviors that dis¬ 
rupt the flow of classroom activities and routines. They gen¬ 
erally require adult intervention to prevent physical harm to 
people or materials/property or to continue with classroom 
routines and activities. Examples of high-intensity chal¬ 
lenging behavior were property destruction, throwing 
objects, physical aggression with a peer, and tantrums. 

An earlier version (Hemmeter et al., 2008b) of the TPOT 
(Fox et al., 2014) was used to measure global changes in 
teachers’ fidelity of implementation of Pyramid Model 
practices over the course of the study. The TPOT is a mea¬ 
sure of teachers’ fidelity of implementation of Pyramid 
Model practices. It is administered by conducting an 
approximately 2-hr observation and 15- to 20-min inter¬ 
view with teachers. The version of the TPOT used in the 
present study had 108 indicators organized under 15 key 
Pyramid Model practice items. Indicators are scored either 


yes (practice was observed or reported to be implemented 
during the interview) or no (practice was not observed or 
reported to be implemented during the interview). In addi¬ 
tion to key practice items, 16 red flags and 7 environmental 
arrangement indicators were included on this version of the 
TPOT. Red flags are practices that are inconsistent with 
Pyramid Model practices. An example would be reprimand¬ 
ing children for expressing their emotions, which is incom¬ 
patible with the key practice item related to teaching 
children to express emotions. The TPOT was used 4 times 
during the study: prior to baseline and after intervention on 
each set of targeted practices for each teacher. Teachers’ 
performance on the TPOT was analyzed by calculating the 
percentage of environmental items present, the percentage 
of indicators present for each of 15 key practice items, and 
the percentage of red flags not present. 

Although the TPOT and CCBOT data were collected by 
data collectors, checklist data were collected by the coaches. 
Checklist data and other information from the observations 
were necessary for the coach to effectively provide feed¬ 
back. If the coach did not collect the primary data, it would 
have been necessary to schedule additional observations to 
allow the coach to gather information needed for coaching 
sessions. However, because the coach served as the primary 
data collector for her teacher, it was important to measure if 
the teacher’s changes in behaviors related to the interven¬ 
tion were dependent on the coach’s presence in the class¬ 
room. Thus, a different data collector observed and collected 
data periodically throughout the intervention phase for each 
targeted practice. The teacher was unaware of the purpose 
of these observations, and the coach was not present during 
these observations. These data will hereafter be referred to 
as alternate observer checks. In addition, interobserver 
agreement (IOA) data were collected on at least 33% of the 
observation sessions to ensure that the coach’s data were 
reliable. 

Experimental Design 

A multiple probe design across sets of practices, replicated 
across teachers, was used to evaluate the effectiveness of 
coaching with performance feedback on teachers’ use of 
Pyramid Model practices. In multiple probe designs, exper¬ 
imental control is demonstrated through the staggered intro¬ 
duction of the independent variable to different behaviors, 
with immediate changes in behavior only after introduction 
of the independent variable (Gast & Ledford, 2010). The 
intervention involved two phases for each practice: (a) 
baseline during which no feedback was provided by the 
coach and (b) intervention during which the coach used 
coaching and performance feedback targeted around the 
current practice until the teacher implemented the indica¬ 
tors at criterion levels. Each tier of the multiple probe design 
represented a different set of targeted practices. 
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General Procedures 

After teachers consented to participate in the study, they 
were assigned a coach. The teachers and coaches met at a 
prestudy kick-off event, during which the study timeline 
and procedures were explained to teachers and the teachers 
and coaches met one another. After the study kick-off, 
TPOT data were collected. Then, each coach conducted an 
informal observation in her teacher’s classroom to familiar¬ 
ize herself with the classroom. After those observations, 
each coach met with her teacher to set goals for coaching. 
During this meeting, the coach described the study and the 
coaching process and helped the teacher identify three goals 
to target during the study. Each potential goal was related to 
a Pyramid Model set of practices (e.g., providing balanced 
and predictable schedules and routines, teaching children 
classroom behavior expectations, promoting emotional lit¬ 
eracy, and teaching social problem-solving) and a corre¬ 
sponding checklist. The coach used the pre-study TPOT 
data, as well as her own observation notes, to narrow the 
field of possible goals/checklists to ones that would most 
benefit the teacher and to help the teacher select action plan 
goals. The coach provided data from the TPOT observation 
to the teacher to guide the selection of action plan goals. 
The teacher was encouraged to choose goals that were foun¬ 
dational and in need of improvement (based on the TPOT 
data). In addition, the teacher was encouraged to choose a 
goal related to targeted social-emotional support that was of 
strong interest to the teacher. Bianca’s goals were as fol¬ 
lows: (a) schedules and routines (targeted practices Set 1), 
(b) behavior expectations (targeted practices Set 2), and (c) 
problem-solving (targeted practices Set 3). Kendra’s goals 
were as follows: (a) behavior expectations (targeted prac¬ 
tices Set 1), (b) schedules and routines (targeted practices 
Set 2), and (c) emotional literacy (targeted practices Set 3). 
Susan’s goals were as follows: (a) behavior expectations 
(targeted practices Set 1), (b) schedules and routines (tar¬ 
geted practices Set 2), and (c) problem-solving (targeted 
practices Set 3). The order in which goals were addressed 
were based on whether the practices were foundational or 
targeted, as well as what order seemed most logical and 
appropriate, given the teacher’s individual classroom and 
child needs. After determining the goals and the order in 
which to address them, baseline data collection began. 

Baseline Procedures 

Each teacher had a minimum of three baseline observations. 
Baseline observations took place during the target activities 
that had been determined during the goal setting meeting 
with the teacher. These typically included a teacher-directed 
large-group activity, a transition, and center time. 

In baseline observations, the coach collected data on the 
three checklists that corresponded to the teacher’s action 


plan goals. The coach did not meet with the teacher or pro¬ 
vide coaching during the period of time when the baseline 
observations were being conducted. At the end of the initial 
baseline observations, CCBOT data were collected in each 
classroom. 

While intervention was being provided for the first set of 
targeted practices, baseline probe data were collected on the 
other sets of targeted practices. Probe data collection on the 
practices in which teachers were not currently being trained 
occurred once per week and in three consecutive observa¬ 
tions before phase changes. Intervention in the first set of 
targeted practices occurred only after stable baseline data 
were observed. Intervention in the second set of targeted 
practices occurred only after the teacher reached the crite¬ 
rion in the first set of targeted practices and had stable base¬ 
line data for the second set of targeted practices. Intervention 
in the third set of targeted practices occurred only after the 
teacher reached the criterion in the second set of targeted 
practices and had stable baseline data for third set of tar¬ 
geted practices. 

Intervention Procedures 

After baseline data were stable, the intervention began. The 
intervention procedures used in this study were consistent 
with our previous work (e.g., Fox et al., 2011; Hemmeter 
et al., 2011). For each set of targeted practices, the interven¬ 
tion began with a session that included training and action 
plan refinement. Although goals were selected prior to 
baseline so they could be measured during baseline, spe¬ 
cific action plans were not developed until the beginning of 
intervention for each set of target practices. The coach pro¬ 
vided a short (30-60 min) training on the practices related 
to the action plan goal/checklist. Training included a stan¬ 
dardized PowerPoint presentation, video examples, and dis¬ 
cussions on how to individualize the practices to the 
teacher’s classroom. The coach also provided four imple¬ 
mentation guides, each focusing on Pyramid Model prac¬ 
tices. The coach and teacher worked together to develop 
steps for meeting the teacher’s first action plan goal. The 
coach and teacher decided on the order in which steps 
would be addressed, a timeline for implementing the steps, 
and materials needed to implement each step. As the teacher 
worked on action plan steps, the coach provided the teacher 
with implementation materials related to the teacher’s 
action plan, such as problem-solving visuals and center 
choice systems. In addition to the initial trainings, it was 
sometimes necessary to provide “booster” trainings. These 
booster trainings occurred after school breaks of at least 2 
weeks (due to scheduled holiday breaks and/or snow days). 
Booster trainings involved reviewing relevant content from 
the first training, as well as the teacher’s action plan. The 
use of booster training is shown in Figures 1 to 3 where a 
“B” is present. 
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Figure I. Bianca’s acquisition, generalization, and maintenance of targeted practices. 


After each training and action planning session, the 
coach conducted live coaching observations, debriefing 
meetings, and email feedback. In addition to the data collec¬ 
tion observations described earlier, the coach conducted a 
coaching observation once a week without collecting spe¬ 
cific data. The purpose of these observations was to provide 
direct coaching (e.g., prompting the teacher, providing 
descriptive praise following the teacher’s use of practices, 
etc.). It was not possible to collect data and provide live 
coaching during the same observations. 

All observations were followed by either live or email 
feedback. Feedback was delivered 3 times per week. 
Approximately 70% of observations were followed by live 
debriefing meetings (approximately twice per week), and 
approximately 30% of sessions were followed by email 
feedback (approximately once per week), except in weeks 
in which scheduling difficulties prohibited this. The 
debriefing meetings and email feedback were based on 
data and notes collected during the checklist data collec¬ 
tion observation or during the live coaching observation. 
Debriefing meetings were individualized to meet the needs 
of the teacher, but they all followed the same general for¬ 
mat, which is consistent with the protocol used in previous 
studies (e.g., Fox et al., 2011; Hemmeter et ah, 2011). The 


coach (a) checked in with the teacher to see whether there 
was anything she would like to share about her classroom, 
(b) reviewed the action plan goal, (c) shared data on the 
action plan goal, (d) provided supportive feedback, (e) 
provided constructive feedback, (f) provided additional 
resources, (g) discussed scheduling upcoming observa¬ 
tions and meetings, and (h) asked the teacher if she had 
questions or concerns. Email feedback followed the same 
protocol, with adjustments to accommodate the differ¬ 
ences between face-to-face and email communication 
(e.g., the inclusion of the graph of the teacher’s data as an 
email attachment). 

The teacher received coaching around her current action 
plan goal until she reached the criterion, which was 80% of 
checklist indicators for three consecutive data collection 
observations. The checklist data were shared with the 
teacher as part of the coaching process, and the teacher was 
aware of the criterion. When providing coaching to the 
teacher, the coach only provided feedback on the current 
action plan goal/set of practices. She did not provide 
instruction or coaching on topics relevant to future or pre¬ 
vious action plan goals. If the teacher’s data dropped below 
criterion on previously addressed action plan goals, the 
coach provided a reminder to continue using the previous 
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Figure 2. Kendra’s acquisition, generalization, and maintenance of targeted practices. 


practices but provided no other feedback. The coach docu¬ 
mented the occurrence and content of these reminders. 

A variety of strategies were used in coaching observa¬ 
tions, debriefing meetings, and email feedback, such as (a) 
providing materials, (b) modeling, (c) helping in the class¬ 
room, (d) problem-solving, (e) reflective conversation, (f) 
environmental arrangement, (g) side-by-side verbal or ges¬ 
tural support, (h) goal setting and planning, and (i) graphing. 

Generalization Procedures 

Generalization across activities was measured periodically 
for all targeted practices and throughout all phases of the 
study using the checklists. The coach and teacher identified 
an additional time of the day for the coach to observe, 
which was afternoon in two classrooms (Kendra and 
Susan) and after lunch in the late morning in one classroom 
(Bianca). The teacher did not receive coaching about using 
the practices during this time of the day. Because of class¬ 
room scheduling issues (e.g., lunch, nap, playground), the 
generalization activities were often quite different than the 
target activities. For example, in one classroom, general¬ 
ization across activities was measured during “quiet cen¬ 
ters,” which occurred directly after rest time, and “wrap-up 
circle.” Although these activities were similar in 
name to the activities on which the teacher was coached 


(i.e., centers and large group), in practice, they were quite 
different. In addition, some checklist indicators could not 
be evaluated during the generalization activities and were 
scored as “no opportunity.” For example, one indicator in 
the schedules and routines checklist was the presence of 
both large- and small-group activities. In some classrooms, 
this was not possible due to the limitations of the schedule 
(i.e., small groups did not occur in the afternoon). The 
teacher was unaware of the purpose of the generalization 
observations. 

Maintenance Procedures 

After the teacher reached the criterion on an action plan 
goal, coaching was not delivered on that goal but data contin¬ 
ued to be collected intermittently. During this time, the coach 
did not provide feedback on the action plan goal but could 
provide reminders about continuing to use the practices when 
there was a drop in the teacher’s performance. These remind¬ 
ers differed from the booster sessions in that the reminders 
were simply a reminder “to continue to use the practices.” 
The reminders did not include a review of the practices or 
additional information about the practices as was done in 
the booster sessions. The frequency and use of reminders 
varied across teachers and can be seen in Figures 1 to 3 
where a “R” is present. 
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Figure 3. Susan’s acquisition, generalization, and maintenance of targeted practices. 


IOA 

Before the study began, data collectors were trained on each 
tool and practiced using each tool in non-participating 
classrooms. They were required to be reliable on each tool 
prior to collecting data for the study. For teacher checklists, 
each data collector was required to complete two observa¬ 
tions (paired with two different observers) at 80% reliability 
for each checklist to be considered reliable. To be consid¬ 
ered reliable on classroom and child measures (i.e., TPOT 
and CCBOT), each data collector had to complete three 
observations with an already trained data collector, with at 
least 80% agreement on the measure being used. 

During the study, IOA data were collected for all teacher, 
classroom, and child measures. At least 30% of observations 
using each measure were conducted with a primary and reli¬ 
ability data collector. The percentage agreement between the 
two data collectors was calculated using a point-by-point for¬ 
mula: The number of agreements divided by the number of 
agreements plus disagreements was multiplied by 100. 

Procedural Fidelity 

Procedural fidelity data were collected on at least 20% of 
each type of coaching session for each coach (i.e., goal 


setting, training and action planning, debriefing, email, 
closing). All coaching sessions were audio recorded, and all 
coaching emails were saved. Coaching sessions and emails 
were randomly selected to be reviewed by a procedural 
fidelity data collector. The data collector used a checklist 
when listening to the audio recordings and viewing the 
emails to determine if the coach followed the protocol for 
each type of session. Examples of procedural fidelity items 
include reviewing the action plan goal, providing positive 
support feedback, and providing constructive feedback. 
Procedural fidelity percentages were calculated by dividing 
the number of items present by the number of items possi¬ 
ble and multiplying by 100. 

In addition, to ensure procedural fidelity was completed 
reliably, another data collector independently completed the 
procedural fidelity checklists for at least 20% of all sessions 
that were reviewed for procedural fidelity. IOA between the 
two procedural fidelity data collectors was calculated using 
point-by-point agreement. 

Social Validity 

At the conclusion of the study, each teacher completed a 
social validity questionnaire. Using a 6-point, Likert-type 
scale (1 = strongly agree to 6 = strongly disagree), teachers 
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rated the following: usefulness of components of the coach¬ 
ing framework, usefulness of various coaching strategies, 
adequacy of the dosage of coaching, relationship with 
coach, and sustainability of practices. Teachers were not 
required to put their name on the questionnaire and were 
given the option to return the completed surveys via U.S. 
mail to the first author. 

Results 

The results will be described as follows: (a) IOA, (b) proce¬ 
dural fidelity, (c) acquisition of targeted practices, (d) alter¬ 
nate observer checks, (e) generalization of targeted 
practices, (f) maintenance of targeted practices, (g) class¬ 
room-wide challenging behavior, (h) TPOT, and (i) social 
validity. 

IOA 

IOA was calculated for each data collection measure. IOA 
data were collected in 41.28% of checklist observations, 
33.33% of generalization checklist observations, 41.67% of 
TPOT observations, and 41.67% of CCBOT observations. 
For the primary measure, the checklists, the mean IOA was 
93.45%, with a range of 56% to 100%. For generalization 
sessions, IOA on the checklists had a mean of 97.15%, with 
a range of 80% to 100%. The mean reliability on the TPOT 
was 87.98%, with a range of 84.40% to 90.15%. Mean 
agreement on the CCBOT was 99.71%, with a range of 
98.9% to 100%. 

Procedural Fidelity 

Procedural fidelity was collected on 44.90% of live coach¬ 
ing sessions and 40.74% of emails. Fidelity percentages 
were high across interventionists and session types. The 
mean fidelity for live coaching sessions was 99.43%, with a 
range of 90.9% to 100%. The mean fidelity for emails was 
100%. Reliability on fidelity was measured in 31.88% of 
the sessions coded for fidelity, with 100% agreement. 

Acquisition of Targeted Practices 

The first three research questions relate to teachers’ acquisi¬ 
tion of targeted Pyramid Model practices, generalization of 
these practices to activities other than those in which they 
were coached, and maintenance of practices. Data from the 
Pyramid Model practice checklists were used to answer 
these questions. The metric used was percentage of indica¬ 
tors present on the checklist. Acquisition of the sets of tar¬ 
geted practices (i.e., percentage of checklist items for a 
goal) is indicated in Figures 1 to 3 by the closed circles. 

Bianca’s baseline data were low and stable for all three 
sets of practices. After coaching began on her use of 


schedules and routines, there was a significant shift in level, 
as can be seen in Figure 1. The double hash marks in the 
abscissa of the graph indicate school breaks of more than 2 
weeks. There was a delay between the end of baseline and 
intervention data collection because there was no school 
due to unanticipated snow days and winter break. No coach¬ 
ing or data collection was possible during this break. This 
break was followed by a booster training on the first set of 
targeted practices. After Bianca’s intervention data for the 
first set of targeted practices were stable (at least three con¬ 
secutive data points at or above the criterion of 80% of 
checklist indicators), coaching began on her second action 
set of targeted practices (i.e., behavior expectations). There 
was an immediate and significant shift in level for this set of 
practices. After intervention data were stable, coaching 
began on Bianca’s third action set of targeted practices (i.e., 
problem-solving). There was an immediate and significant 
shift in level that stabilized before ending intervention. 

Kendra’s baseline data were low and stable for her first 
set of targeted practices (i.e., behavior expectations) as can 
be seen in Figure 2. After coaching began on this set of tar¬ 
geted practices, there was an immediate shift in the level of 
the data. After Kendra’s intervention data for the first set of 
targeted practices were stable, coaching began on her sec¬ 
ond set of targeted practices (i.e., schedules and routines). 
Although baseline data for this set of targeted practices 
were variable, there was an immediate and significant shift 
in level for this set of practices. After these data were stable 
and at criterion, coaching began on her third set of targeted 
practices (i.e., emotional literacy), which resulted in an 
immediate and significant shift in level of the data. 

Susan’s baseline data were low and stable for all three 
sets of practices that were targeted for coaching, as can be 
seen in Figure 3. After coaching began on the first set of 
practices (i.e., behavior expectations), there was a signifi¬ 
cant shift in the level of the data. As with Bianca, this shift 
was delayed due to school breaks, which is indicated by the 
double hash marks in the abscissa of her graphs. This break 
was followed by a booster training on the first set of targeted 
practices. Once these data were stable, coaching began on 
her second set of targeted practices (i.e., schedules and rou¬ 
tines). There was an immediate and significant change in 
level for this set of practices. A similar pattern was seen for 
her third set of targeted practices (i.e., problem-solving). 

Alternate Observer Checks 

Alternate observer checks were conducted to ensure that the 
teacher used targeted practices when her coach was not 
present and collecting data. A data collector other than the 
coach collected data during these observations. During the 
alternate observer checks, each teacher demonstrated prac¬ 
tices she was being coached on or had already acquired 
even when her coach was not present at about the same 
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Table 1. 

TPOT and CCBOT Data. 






Teacher 

Time 


TPOT data 



CCBOT data 


Percentage of 
indicators (%) 

Number of 
environmental 
items 

Number of 
red flags 

Percentage of intervals with challenging behavior 
present 

Total" (%) 

Low intensity (%) 

High intensity (%) 

Bianca 

Baseline 

28.70 

3 

5 

25.74 

23.33 

2.96 


After Set 1 

43.52 

4 

0 

16.67 

12.04 

4.81 


After Set 2 

39.81 

5 

1 

15.37 

13.33 

2.04 


After Set 3 

49.07 

7 

1 

I0.56 b 

I0.37 b 

0.19 b 

Kendra 

Baseline 

41.67 

5 

2 

24.26 

17.59 

9.44 


After Set 1 

44.44 

7 

2 

21.48 

12.78 

8.70 


After Set 2 

58.33 

7 

0 

l.l 1 

0.93 

0.19 


After Set 3 

44.44 

6 

2 

2.78 

1.30 

1.48 

Susan 

Baseline 

34.26 

6 

5 

4.44 

0.74 

3.70 


After Set 1 

46.30 

7 

3 

13.15 

1 1.30 

1.85 


After Set 2 

38.89 

7 

4 

9.81 

7.59 

2.41 


After Set 3 

60.19 

7 

0 

21.85 

15.93 

6.11 


Note. TPOT = Teaching Pyramid Observation Tool; CCBOT = Class-Wide Challenging Behavior Observation Tool. 

a Some intervals had both low- and high-intensity problem behavior. b There was one missing data session for Bianca in the final data collection point; 
therefore, only one 45-min session is presented. 


level (i.e., percentage of checklist items present) as when 
her coach was present and collecting data. This can be seen 
in Figures 1 to 3 by examining the open triangles. 

Generalization 

Generalization across activities is indicated in Figures 1 to 
3 by the closed diamonds. There were mixed results across 
teachers. Bianca demonstrated strong results for generaliza¬ 
tion across activities. After she demonstrated acquisition of 
a set of practices, her generalization data indicate that she 
used those practices during activities in which she received 
no coaching at levels above what was observed during base¬ 
line. She demonstrated the practices around criterion levels 
(80% of checklist items) during the generalization activi¬ 
ties. Kendra (see Figure 2) demonstrated inconsistent results 
for generalization across activities. After she acquired a set 
of practices, she demonstrated the practices in other activi¬ 
ties at higher levels than was observed during baseline for 
two of the skills (set of targeted practices 1 and 3). Flowever, 
these levels were significantly lower than those observed in 
the intervention activities. As can be seen in Figure 3, Susan 
used the practices during the generalization activities above 
baseline levels but not near criterion levels. 

Maintenance 

Maintenance data are represented in Figures 1 to 3 by 
examining the closed circles in each graph after interven¬ 
tion ended on that goal. After coaching on a goal ended, 


intermittent data were collected. Kendra maintained all 
practices at criterion levels, but Bianca and Susan both 
needed reminders to continue using practices they had 
already acquired. These reminders are denoted on their 
graphs by “R” under the abscissa at the time points when 
these reminders were given. Bianca received one reminder 
related to her first action plan goal and two reminders related 
to her second action plan goal. Susan received three remind¬ 
ers related to her first action plan goal and one reminder 
related to her second action plan goal. These reminders 
were simply brief statements to the teacher to remind her to 
continue using practices; coaching was not provided. With 
these reminders, both Bianca and Susan maintained their 
use of practices that were targeted for coaching even when 
coaching ended. Because of the end of the school year, no 
maintenance data were collected on third set of targeted 
practices for any of the teachers. 

Classroom-Wide Challenging Behavior 

The fourth research question addressed the level of class¬ 
room-wide challenging behavior when Pyramid Model 
practices were implemented with fidelity. The results of the 
CCBOT at each point in data collection are presented in 
Table 1. For both Bianca and Kendra, there was a decrease 
in classroom-wide challenging behavior from baseline to 
intervention. In Susan’s classroom, levels of challenging 
behavior were quite variable, with the lowest level observed 
during baseline and the highest level observed at the end of 
intervention. 
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Overall Implementation of the Pyramid Model 

The fifth research question addressed whether overall imple¬ 
mentation of the Pyramid Model (i.e., TPOT scores) 
increased after teachers received training and coaching on 
specific Pyramid Model practices. TPOT data are presented 
in Table 1 by total percentage of indicators on the observa¬ 
tion and interview questions, number of environmental 
items present, and number of red flags present. High scores 
are desired for indicators and environmental items, and low 
scores are desired for red flags. There was an increase in 
the percentage of indicators on the TPOT from baseline to 
final data collection by approximately 21% for Bianca, 3% 
for Kendra, and 26% for Susan. In all classrooms, the num¬ 
ber of environmental items increased. The number of red 
flags decreased for Bianca and Susan but stayed the same for 
Kendra. 

Social Validity 

The final research question related to social validity and 
was assessed by examining teachers’ perspectives on coach¬ 
ing and the sustainability of the Pyramid Model practices. 
Overall, the teachers found the coaching strategies (e.g., 
observation, feedback, action planning) useful. Although 
some teachers found coaches’ observations stressful, all 
agreed that the strategy was useful. Each teacher strongly 
agreed she had a good working relationship with her coach. 
Teachers agreed that they would continue to use the Pyramid 
Model practices in the future. 

Discussion 

The primary purpose of this study was to investigate the 
effects of professional development intervention that 
included training and coaching with performance feedback 
on teachers’ use of Pyramid Model practices and whether 
teachers generalized and maintained their use of practices. 
All teachers in this study increased their use of targeted 
teaching practices. Thus, there were nine demonstrations/ 
replications across the three teachers, establishing a func¬ 
tional relation. Although it was not possible to establish a 
functional relation in regard to the teachers’ generalization 
and maintenance of practices (due to the design of the 
study), there were promising results for both. All teachers 
maintained their practices with only periodic reminders to 
use the practices. The results related to generalization to 
other activities were mixed, with strong generalization 
demonstrated by Bianca and more limited results for the 
other two teachers. 

The second purpose of this study was to measure the 
impact of teachers’ use of practices on a measure of class¬ 
room-wide challenging behavior. These data were mixed 
across teachers. For Bianca and Kendra, there was a 


decrease in challenging behavior from baseline to the end 
of the intervention. For Susan, there was no discernible 
pattern in the data on classroom-wide challenging behav¬ 
ior. There was an increase in challenging behavior after 
baseline, which was due primarily to a new student with 
significant challenging behavior joining the classroom. 

The third purpose was to measure whether teachers’ 
overall TPOT scores increased when they received coach¬ 
ing on specific Pyramid Model practices. All teachers’ 
TPOT scores improved after receiving coaching in specific 
practices. This indicates that scores on a global measure can 
be positively affected when focusing only on a few key 
practices. 

Limitations 

There were five limitations to this study. The first limita¬ 
tion is that each coach collected the primary data in her 
teacher’s classroom. This could have introduced bias in 
data collection. The IOA data address this concern, and 
percentages were high across teachers. The coach serving 
as the primary data collector was also problematic in that 
the teachers might have only used the practices in the pres¬ 
ence of their coach. This was addressed by having the 
alternate observer checks. These data demonstrated that 
the teachers used practices when the coach was not present 
at levels similar to those observed when the coach was 
present. 

The second limitation is that generalization across activ¬ 
ities was limited in part because of issues related to when 
the data were collected. This is because there were few 
opportunities to observe many of the targeted practices at 
times other than during the primary data collection (i.e., 
morning). Because of the numerous routine activities nec¬ 
essary in preschool classrooms (e.g., lunch, nap, bathroom) 
and the variability in schedules in the afternoon, this was an 
unavoidable limitation. Nonetheless, this limited the evi¬ 
dence of teachers’ generalization of practices. 

The third limitation relates to the teachers’ maintenance 
of practices. The school year ended before maintenance 
data could be collected in the third set of targeted practices. 
Furthermore, the end of the school year prevented us from 
observing whether the teachers would have continued to 
use the first and second sets of targeted practices without 
the occasional reminders. 

The fourth limitation relates to the design of the study. 
Because the coaching intervention was implemented across 
practices, the coaches only coached on one set of practices 
at a time and did not address other practices even when they 
were linked to the set of targeted practices. For example, 
when working on schedules and routines, the coaches did 
not address transitions even though those practices could 
clearly be related. This might have decreased the likelihood 
of seeing more significant global changes in teacher 
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practices. Outside the confines of a study, coaching would 
not have these restraints. 

A final limitation was the inconclusive findings related to 

the effects of the intervention on children’s challenging 
behavior. There are a number of possible reasons why these 
findings were not more robust. First, it may be that the spe¬ 
cific practices that were targeted (e.g., schedules and rou¬ 
tines, problem-solving, emotional literacy) were more likely 
to affect children’s engagement and pro-social behaviors 
which were not measured in the current study. Second, it is 

possible that these findings were limited because of the dif¬ 
ficulty in measuring incidences of classroom-wide challeng¬ 
ing behavior including (a) the influence of a single child’s 
behavior on the classroom-wide measure, (b) the lack of sta¬ 

bility in young children’s behavior, and (c) the influence of 
the type and quality of activities occurring in the classroom 
when the observation is conducted. It is also possible that the 

“dose” of the teachers’ implementation of the practices was 
not sufficient to affect changes in children’s behavior. 

Finally, coaching teachers around classroom-wide 
behavior support practices may not be sufficient for reduc¬ 
ing the challenging behavior of individual children. In 
research studies in which teachers have been effective in 
implementing interventions that result in reductions of chal¬ 
lenging behavior, teachers have been coached on the imple¬ 
mentation of behavior intervention practices specifically 
designed for preventing and responding to an individual 
child’s behavior (e.g., Blair, Fox, & Lentini, 2010; Duda, 
Dunlap, Fox, Lentini, & Clarke, 2004; Smith, Lewis, & 
Stormont, 2011). In the current study, teachers were trained 
to use practices with all children in the classroom, and no 
data were collected on the use of practices in relationship to 
individual children with behavior challenges. Future studies 
might include a coaching component to ensure that prac¬ 
tices are being used with sufficient dose, frequency, and in 
a manner that addresses the challenging behavior of indi¬ 
vidual children. 

Implications for Research and Practice 

Implications for research and practice relate to how to 
ensure generalization and maintenance of practices, the 
intensity of coaching, and the targeted nature of the coach¬ 
ing. In this study, the results related to generalization were 
mixed. Teachers did not consistently use targeted practices 
in untrained activities. This indicates that when coaching 
teachers, it might be necessary to program for generaliza¬ 
tion by coaching during different times of the day and in 
different activities and explicitly coaching teachers on how 
to use the practices in a variety of activities. The mainte¬ 
nance results indicate that teachers can continue to imple¬ 
ment practices in the absence of coaching. Two teachers 
needed only brief reminders to continue using the practices, 


and one did not need any reminders. The use of reminders 
could be translated into practice through the use of periodic 
“check-ins” with teachers to ensure their continued use of 
targeted practices. 

Intensive coaching was needed for teachers to imple¬ 
ment practices at or above criterion. Coaching was provided 
to the teacher 2 to 3 times per week in person or via email. 
Across all three sets of targeted practices, Bianca had 26 
live sessions and 10 emails, Kendra had 16 live sessions and 
8 emails, and Susan had 21 live sessions and 9 emails. This 
intensity of coaching should be considered when planning 
future research around coaching and when making deci¬ 
sions about how to allocate professional development 
resources within a program. It is possible that this dosage 
might be reduced if a program-wide model was imple¬ 
mented where teachers received other types of support. It is 
also possible that teachers working on the same practices 
could be coached in groups. If an entire program was imple¬ 
menting the Pyramid Model at a high level of fidelity, there 
would be more implementation supports for teachers in the 
form of administrative commitment, fewer competing 
demands, peer support, family engagement, and support for 
practices at all levels of the Pyramid Model (Fox & 
Hemmeter, 2009). 

As resources for professional development become more 
limited, efficient coaching models are needed. Although 
this study provided evidence that an individualized coach¬ 
ing model was effective, it is important to study how coach¬ 
ing can be provided more efficiently without compromising 
the integrity and effectiveness of the model. Furthermore, 
research is needed on how to train and support coaches in 
their use of effective coaching models. 
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